Skip to content

feat(storage): pluggable dump storage backends#7

Merged
nixrajput merged 3 commits into
mainfrom
feat/phase-g-cloud-storage
Jun 25, 2026
Merged

feat(storage): pluggable dump storage backends#7
nixrajput merged 3 commits into
mainfrom
feat/phase-g-cloud-storage

Conversation

@nixrajput

@nixrajput nixrajput commented Jun 23, 2026

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • New Features
    • Added pluggable storage backend support (Phase G). Dumps can now be stored in S3 or S3-compatible services via a new storage: configuration block (bucket, prefix, region, endpoint).
    • Improved cross-backend SHA-256 integrity verification using consistent envelope/body hashing.
  • Documentation
    • Added STORAGE.md covering storage configuration, guarantees, and verification behavior.
  • Bug Fixes / Improvements
    • Backup, restore, verify, and dump-related UI/CLI operations now use context for better cancellation/timeout handling.
  • Tests
    • Added shared store conformance tests and S3 integration coverage using MinIO.

- Add internal/storage leaf: a backend-neutral Store interface
  (Put/Get/Delete/List/Stat, context-aware) with a shared ErrNotFound
  sentinel and a RunStoreSuite contract test every backend runs.
- Ship two backends: local (temp-write+rename atomic publish, keys map
  verbatim to file names so existing catalogs need no migration) and s3
  (AWS SDK v2 transfer-manager streaming upload, atomic on completion,
  path-style for S3-compatible endpoints like MinIO/R2).
- Refactor dumps.Catalog to hold a Store and address dumps by key
  instead of filesystem paths; drop Path/MetaPath/Root. Catalog methods
  are now context-aware. backup/restore/verify/dumps stream through the
  store; SHA-256 integrity stays app-side over envelope++body, so a dump
  verifies identically whether local or in S3.
- Add a storage: config block (type local|s3, bucket, prefix, region,
  endpoint) with fail-fast validation; S3 credentials resolve from the
  standard AWS chain, never stored in config.
- Cover the local backend with the contract suite plus atomic-publish
  and key-layout unit tests; cover the s3 backend with the same contract
  suite against MinIO via testcontainers under the integration tag.
- Document the feature in docs/STORAGE.md; update README and CHANGELOG.
@nixrajput nixrajput self-assigned this Jun 23, 2026
@coderabbitai

coderabbitai Bot commented Jun 23, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@nixrajput, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 48 minutes and 4 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 57dd8f9e-75bc-4a21-9a16-48d751ab4765

📥 Commits

Reviewing files that changed from the base of the PR and between b5be96b and 54aaacc.

📒 Files selected for processing (2)
  • internal/storage/local.go
  • internal/storage/local_test.go
📝 Walkthrough

Walkthrough

Phase G adds a backend-neutral storage.Store interface with local and S3 implementations, refactors the dump catalog to use storage keys and contexts, wires the new configuration through startup and commands, and updates backup, restore, verify, tests, and documentation.

Changes

Phase G: Pluggable Storage Backend for Dump Catalog

Layer / File(s) Summary
Store interface, ErrNotFound, and RunStoreSuite contract harness
internal/storage/store.go, internal/storage/suite.go
Defines the Store interface contract, the ErrNotFound sentinel, and the RunStoreSuite harness covering round-trip, missing-key, stat, delete, list, overwrite, and cancellation behavior.
Local filesystem backend with atomic publish
internal/storage/local.go, internal/storage/local_test.go
Implements a local store with key validation, atomic temp-file publishing, context-aware copying, idempotent delete, listing, and stat behavior; tests cover the contract suite, failed publish cleanup, and key layout.
S3 backend, not-found detection, and MinIO integration test
internal/storage/s3.go, internal/storage/s3_integration_test.go, go.mod
Adds S3-backed storage with AWS SDK v2, multipart upload, paginated listing, S3-compatible not-found handling, a MinIO contract test, and the required module dependencies.
StorageConfig type, Validate(), and Load() integration
internal/config/config.go, internal/config/storage_test.go
Adds StorageConfig, validates storage type and bucket requirements, and wires validation into config loading with table-driven tests.
Catalog: storage-backed, context-aware, key-based
internal/dumps/catalog.go, internal/dumps/chain.go, internal/dumps/prune.go, internal/dumps/catalog_test.go, internal/dumps/chain_test.go
Replaces filesystem-backed catalog access with a storage backend, adds key-based dump and metadata operations, and updates chain/prune helpers and tests to use context.
Backup, restore, verify, CLI, and TUI wiring
internal/app/backup.go, internal/app/restore.go, internal/app/verify.go, internal/cli/root.go, internal/cli/dumps.go, internal/tui/..., internal/app/*_test.go
Reworks backup streaming and checksum counting, switches restore and verify to catalog-backed reads, selects the storage backend at startup, and threads contexts through CLI and TUI dump operations.
Documentation: STORAGE.md, README, and CHANGELOG
docs/STORAGE.md, README.md, CHANGELOG.md
Adds storage documentation, updates README status and configuration guidance, and records the Phase G storage work in the changelog.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • nixrajput/siphon#1: Introduced the earlier dump catalog implementation that this PR refactors into a storage-backed, context-aware design.
  • nixrajput/siphon#6: Modified backup and restore paths that are further reworked here to use PutDump, OpenDump, and the new storage-backed catalog.

Poem

🐇 I hopped through keys from root to cloud,
With temp-file paths now trimmed and shrouded.
Local and S3 both spring to life,
Hashes stay true through dump and strife.
I nibble configs, then dash away—
Phase G is hopping out today!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 58.82% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: introducing pluggable dump storage backends.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/phase-g-cloud-storage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/config/config.go`:
- Around line 43-50: The Validate method of StorageConfig does not trim
whitespace from the Type and Bucket fields before validation, allowing values
with only whitespace to pass validation and fail later at runtime. At the
beginning of the Validate method, trim whitespace from both s.Type and s.Bucket
fields using strings.TrimSpace, then proceed with the existing validation logic.
This ensures that storage.type and storage.bucket values containing only spaces
are rejected immediately rather than deferring failure to runtime.

In `@internal/dumps/catalog.go`:
- Around line 152-156: The order of deletion operations in the delete method is
incorrect and risks leaving dangling catalog entries on partial failure.
Currently, the code deletes the dump file first using dumpKey(id) and then the
metadata using metaKey(id). Reverse the order of these two Delete calls so that
the metadata is deleted first using metaKey(id), then the dump file is deleted
using dumpKey(id). This ensures that if the second delete fails, the worst case
is an orphaned dump body rather than a catalog entry pointing to a deleted body.
- Around line 136-139: In the List method, the current error handling after
calling ReadMeta skips all errors indiscriminately using continue, which masks
backend failures and context cancellation errors. Instead, differentiate between
error types: only continue for corrupt entry errors that should be skipped by
the user later, but return immediately for system errors like context.Canceled
or other backend failures. Check the error type or message to determine if it
represents a corrupt user entry before deciding whether to continue or propagate
the error.

In `@internal/storage/local_test.go`:
- Around line 38-39: The os.ReadDir call on line 38 ignores the returned error
using underscore, which can mask setup failures and result in misleading
assertion messages when the directory is checked for cleanliness. Capture the
error returned by os.ReadDir instead of discarding it, then add explicit error
handling (typically using t.Fatal or t.Errorf in the test) to properly report if
the ReadDir operation fails before proceeding with the len(entries) != 0
assertion.

In `@internal/storage/local.go`:
- Around line 58-66: The Put method fails to guarantee durable writes as
declared in the interface because it only closes and renames the temporary file
without syncing data to disk. After writing to the temporary file (tmp), call
Sync on the file handle before closing it to ensure all file data is written to
disk. Additionally, after the os.Rename call that moves tmpName to its final
location p, open and sync the parent directory to ensure the metadata changes
from the rename operation are persisted, which will satisfy the durability
guarantee.
- Around line 62-63: The os.Rename call in the Store.Put method fails on Windows
when the destination file already exists, violating the overwrite contract
required by the interface. To fix this, implement a platform-safe atomic replace
strategy by first removing the destination file (using os.Remove on the path p)
before attempting os.Rename, or use a cross-platform helper function that
handles both Windows and Unix-like systems. This ensures that overwriting an
existing key works consistently across all platforms in the CI matrix, including
windows-latest.

In `@internal/storage/store.go`:
- Around line 18-23: The documentation for the ErrNotFound variable incorrectly
states that Get/Stat/Delete all return ErrNotFound when a key is absent, but
this contradicts the interface contract where Delete and Stat are defined as
non-errors for missing items. Update the documentation comment for ErrNotFound
to accurately reflect that only Get returns ErrNotFound for missing keys,
removing the incorrect references to Stat and Delete from the description.

In `@internal/storage/suite.go`:
- Around line 125-126: The io.ReadAll call at line 125 and the Stat call at line
140 are ignoring errors by using blank identifiers, which can mask real backend
failures in the suite tests. Capture the error return values from both
io.ReadAll and Stat operations, and add appropriate error assertions or checks
(such as asserting the error is nil or handling the error case) to ensure these
operations succeed before proceeding with the contract assertions. This will
prevent the tests from passing when there are actual backend failures.

In `@internal/tui/modals/restore.go`:
- Line 30: The call to d.Dumps.List() in the restore modal uses
context.Background() which creates an unbounded context that can block the TUI
indefinitely during remote storage operations (e.g., S3). Replace
context.Background() with a bounded context that has a timeout, or refactor the
NewRestore function to accept a context parameter from the caller and thread
that context through to the List() call. This ensures the storage I/O operation
is cancellable and won't hang the user interface.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 77f7ee94-5658-43e0-9b25-f5cc4e7ead49

📥 Commits

Reviewing files that changed from the base of the PR and between ac91634 and 2f20ebe.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (28)
  • CHANGELOG.md
  • README.md
  • docs/STORAGE.md
  • go.mod
  • internal/app/backup.go
  • internal/app/backup_test.go
  • internal/app/restore.go
  • internal/app/restore_chain_test.go
  • internal/app/roundtrip_test.go
  • internal/app/verify.go
  • internal/cli/dumps.go
  • internal/cli/root.go
  • internal/config/config.go
  • internal/config/storage_test.go
  • internal/dumps/catalog.go
  • internal/dumps/catalog_test.go
  • internal/dumps/chain.go
  • internal/dumps/chain_test.go
  • internal/dumps/prune.go
  • internal/storage/local.go
  • internal/storage/local_test.go
  • internal/storage/s3.go
  • internal/storage/s3_integration_test.go
  • internal/storage/store.go
  • internal/storage/suite.go
  • internal/tui/dashboard_test.go
  • internal/tui/modals/restore.go
  • internal/tui/panels/dumps.go

Comment thread internal/config/config.go
Comment thread internal/dumps/catalog.go
Comment thread internal/dumps/catalog.go Outdated
Comment thread internal/storage/local_test.go Outdated
Comment thread internal/storage/local.go
Comment thread internal/storage/local.go Outdated
Comment thread internal/storage/store.go Outdated
Comment thread internal/storage/suite.go Outdated
Comment thread internal/tui/modals/restore.go Outdated
- Apply post-review fixes to the cloud-storage cycle.
- local Put: fsync the temp file and the parent dir around the rename
  so a dump that survives Put is durable across a crash (the Store
  contract promises durability and this is a backup tool).
- local Put: replace os.Rename with atomicRename, which removes the
  destination and retries where rename-over-existing can fail (Windows),
  honoring the overwrite contract across filesystems.
- Catalog.List: propagate system/cancellation errors instead of
  skipping every ReadMeta failure; only skip CodeUser (corrupt/missing)
  entries, so a backend outage or cancelled context is not hidden as
  partial success.
- Catalog.Delete: remove the meta object first, then the body, so a
  partial failure leaves at worst an invisible orphan body rather than
  a catalog entry pointing at a deleted body (mirrors meta-last write).
- config Validate: trim whitespace on type/bucket so a blank-but-
  nonempty value fails fast instead of at runtime.
- TUI restore modal and dumps panel: bound the catalog List with a 5s
  timeout so a slow/unreachable S3 backend cannot hang the UI.
- store.go: fix the ErrNotFound doc to match the contract (Get returns
  it; Delete/Stat treat a missing key as a non-error).
- Tests: check the previously-ignored ReadDir error in the atomic-
  publish test and the ReadAll/Stat errors in the contract suite.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/storage/local.go`:
- Around line 88-91: The fallback in atomicRename is too broad: it removes dst
after any os.Rename failure, which can delete valid data on unrelated errors.
Update the rename retry logic in atomicRename to only attempt removing dst when
the failure specifically indicates the destination already exists or cannot be
replaced, and otherwise return the original error immediately. Keep the
retry/remove behavior guarded by the existing os.Rename and os.Remove flow so
the overwrite path is only used for the intended case.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4905e6dd-2459-41d7-860e-7a6e9b76b815

📥 Commits

Reviewing files that changed from the base of the PR and between 2f20ebe and b5be96b.

📒 Files selected for processing (8)
  • internal/config/config.go
  • internal/dumps/catalog.go
  • internal/storage/local.go
  • internal/storage/local_test.go
  • internal/storage/store.go
  • internal/storage/suite.go
  • internal/tui/modals/restore.go
  • internal/tui/panels/dumps.go
🚧 Files skipped from review as they are similar to previous changes (5)
  • internal/storage/store.go
  • internal/storage/local_test.go
  • internal/storage/suite.go
  • internal/config/config.go
  • internal/dumps/catalog.go

Comment thread internal/storage/local.go Outdated
- The prior atomicRename removed the destination and retried on ANY
  os.Rename error. A transient failure unrelated to the destination
  existing (I/O glitch, permissions, device busy) would then delete the
  existing valid dump and still fail — destroying the last good copy.
- Guard the remove+retry to fire only when the rename error matches
  fs.ErrExist (the destination-exists case on Windows / some
  filesystems); any other error returns untouched, leaving dst intact.
- Add TestAtomicRename_PreservesDstOnUnrelatedError: a rename from a
  missing source must leave the existing destination unharmed.
@nixrajput nixrajput merged commit 01d23f2 into main Jun 25, 2026
5 checks passed
@nixrajput nixrajput deleted the feat/phase-g-cloud-storage branch June 25, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant